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[NIST—2021—0004; Docket No. 210726-0151] 


BSA | The Software Alliance appreciates the opportunity to provide feedback on the 
National Institute of Standards and Technology’s (NIST) Request for Information 
regarding the development of an Artificial Intelligence Risk Management 
Framework (Al RMF).' BSA is an association of the world's leading enterprise 
software companies that provide businesses in every sector of the economy with 
tools to operate more competitively and innovate more responsibly. As leaders in 
the development of enterprise Al, BSA members have unique insights into the 
technology’s tremendous potential and the policies that can best support the 
responsible use of Al. BSA members recognize that trust is essential to the public’s 
willingness to embrace these new technologies and are therefore strongly 
supportive of NIST’s mission to “cultivate the public’s trust in the design, 
development, use, and evaluation of Al technologies and systems in ways that 
enhance economic security, and improve quality of life.”? 


BSA has long recognized that risk management is a key component of promoting 
trust in Al. For that reason, BSA has advocated for a NIST-led process to develop 





1 86 Fed. Reg. 40810 (July 29, 2021), available at https://www.govinfo.gov/content/pkg/FR- 
2021-07-29/pdf/2021-16176.pdf (hereinafter RFI) 
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an Al risk management framework.? We are therefore extremely encouraged by the 
announcement outlined in the RFI. We write today both to applaud the vision and 
process set forth in the RFI and to provide initial feedback on “how individuals, 
groups and organizations involved with designing, developing, using, or evaluating 
Al systems might be better able to address the full scope of Al risk and how a 
framework for managing Al risks might be constructed.”* 


Our feedback is informed by our recent experience working with BSA member 
companies to develop the BSA Framework to Build Trust in Al (the Framework),° a 
risk management framework for mitigating the potential for unintended bias 
throughout an Al system’s lifecycle. Built on a vast body of research and informed 
by the experience of leading Al developers, the Framework outlines a lifecycle- 
based approach for performing impact assessments to identify risks of Al bias and 
highlights corresponding risk mitigation best practices. 


BSA encourages NIST to consider the BSA Framework as a resource during the 
development of the NIST Al RMF and note that it is consistent with many of the 
“attributes” that the RFI identifies as priorities. For instance, the BSA Framework 
was drafted to be a resource for a “broad audience” of stakeholders — including 
practitioners and senior executives; it is “scalable to organizations of all 
sizes...[and is] platform- and technology-agnostic”; it provides a “risk-based, 
outcome-focused” approach for managing Al risks; it can be integrated into “any 
enterprise's broader risk management strategy and processes”; and it highlights 
“existing standards, guidelines, best practices, methodologies, and tools for 
managing Al risks.” 


Although the BSA Framework is focused on one key risk to Al trust — i.e., 
unintended bias — it was developed and structured in a manner that makes it 
adaptable to the broader set of “characteristics of trust” that NIST seeks to 
address. The BSA Framework is ultimately an assurance-based mechanism that 





3 See, Comments of BSA | The Software Alliance Regarding NIST’s Federal Artificial 
Intelligence Standards Engagement Plan (May 31, 2019), available at 
https://www.bsa.org/files/policy-filings/0610201 9bsasubmissionaistandardsrfi.pdf. 

4 RFI ad 408111. 


5 Confronting Bias: The BSA Framework to Build Trust in Al (June 2021), available at 
https://ai.bsa.org/wp-content/uploads/2021/06/2021bsaaibias.paf. 
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organizations can use to enhance trust in their Al systems through risk 
management processes that promote fairness, transparency, and accountability. 
We anticipate that many of the details and attributes of the BSA Framework will be 
relevant to the NIST’s development of an Al RMF. We highlight three key 
considerations below that will be crucial to the Al RMF’s success. 


|. The NIST Al RMF Should Incorporate a Lifecycle-Based Approach 


Static evaluations of Al models cannot account for all potential issues that may 
arise when Al systems are deployed in the field. Experts agree that Al risk 
management therefore requires a lifecycle approach that includes ongoing 
monitoring by end-users to ensure that the system is operating as intended. As 
noted in Draft NIST Publication 1270, the Al lifecycle is a helpful concept for 
organizing risk management functions because it is already used by 
“[o]rganizations that design and develop Al technology...to keep track of their 
processes and ensure delivery of high-performing functional tools.”* 


While there are numerous articulations of the Al lifecycle, we agree with the 
approach outlined in Draft NIST Publication 1270 for organizing the lifecycle into 
three key phases that are loosely correlated to Pre-Design (i.e., problem 
formulation and training data selection), Design and Development (i.e., modeling 
and validation), and Deployment (i.e., monitoring.) The three-stage formulation of 
the Al lifecycle is a helpful heuristic for identifying key risk management functions 
and mapping them to the appropriate stakeholders. The BSA Framework is 
similarly organized around a three-stage lifecycle that maps both the unique risks 
that can arise and corresponding mitigations that should be considered during the 
“Design” (project conception and data selection), “Development” (engineering and 
validation), and “Deployment” (i.e., use and monitoring) of a high-risk Al system. 
We encourage NIST to incorporate a similar three-stage lifecycle-based approach 
into the organization of the Al Risk Management Framework. 





$ Draft NIST Publication 1270, A Proposal for Identifying and Managing Bias in Artificial 
Intelligence (June 2021) at page 6, available at 
https: //nvipubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270-draft.paf. 
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ll. The NIST Al RMF Should Recognize Risk Management as a Shared 
Responsibility 


Reflecting the inherently dynamic nature of Al systems, the NIST Al RMF should 
account for the array of stakeholders that may play a role in various aspects of a 
system’s design, development, and deployment. Because there is no single model 
of Al development or deployment, it is impossible in the abstract to assign roles or 
delegate specific responsibilities for many risk management functions. In the BSA 
Framework, we highlight that there are at least three sets of stakeholders that may 
bear varying degrees of responsibility for certain aspects of Al 

risk management throughout a system’s lifecycle: 


e Al Developers. Al Developers are organizations responsible for the design 
and development of Al systems. 

s Al Deployers. Al Deployers are the organizations that adopt and use Al 
systems. (If an entity develops its own system, it is both the Al Developer and 
the Al Deployer.) 

e Al End-Users. Al End-Users are the individuals—oftentimes an employee of 
an Al Deployer—who are responsible for overseeing the use of an Al system. 


The appropriate allocation of risk management responsibilities between 
stakeholders will vary depending on the nature of the Al system being developed 
and which party determines the purposes and means by which the underlying 
model is trained. For instance: 


e Universal, Static Model. The Al Developer provides all of its customers (i.e., 
Al Deployers) with a static, pre-trained model. 

o The Al Developer will bear responsibility for most aspects of model 
risk management during the Design and Development stages. Risk 
management functions during the Deployment stage may be shared. 

e Customizable Model. The Al Developer provides a pre-trained model to Al 
Deployers who can customize and/or retrain the model using their own data. 

o While the Al Developer will be responsible for risk management of 
Design decisions, risk management associated with the Development 
and Deployment stages will be a shared responsibility between the Al 
Developer and the Al Deployer. 

e Bespoke Model. The Al Developer trains a bespoke Al model on behalf of 
an Al Deployer using the Al Deployer's data. 

o The bulk of risk management obligations will fall on the Al Deployer. 


As NIST moves forward with the development of an RMF, it will likewise be 
important to consider how the spectrum of Al development models impacts which 
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stakeholder will be best positioned to perform particular risk management 
functions. 


Ill. The NIST Al RMF Should be Informed by Emerging Policy 
Considerations 


BSA strongly supports NIST’s intention to develop the Al RMF in a manner that is 
“law- and regulation-agonistic” so that it can “support organizations” ability to 
operate under applicable domestic and international legal or regulatory regimes.”” 
At the same time, NIST should remain attuned to the global convergence around 
the need for risk-based approaches to Al governance. A cornerstone of such efforts 
relates to the need for organizations that develop and/or deploy high-risk Al 
systems to document their efforts to identify and mitigate potential risks to the 
public. The European Commission's recent introduction of the Artificial Intelligence 
Actis a good case in point. Central to the Artificial Intelligence Act's approach to 
governance is a requirement for providers of high-risk Al systems to implement a 
“quality management system” that documents the “policies, procedures, and 
instructions” to ensure compliance with the Act's other provisions (Article 17), 
including the implementation of a risk management system (Article 9). $ 


The NIST Al RMF can play a helpful role in shaping the development of these 
efforts in a manner that will promote global interoperability. To that end, the Al RMF 
process should be viewed as an opportunity to help identify existing (or lay the 
foundation for new) points of consensus in areas that are likely to be the focus of 
future regulatory requirements around the world. As noted above, one area that is 
particularly ripe for consideration relates to technical documentation related to Al 
risk management and how stakeholders can effectively communicate about their 
respective approaches to identifying and mitigating potential risks. 





7 RFI at 40812. 


$ European Commission, Proposal for a Regulation of the European Parliament and of the 
Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) 
and Amending Certain Union Legislative Acts (COM/2021/206 final), available at https://eur- 
lex.europa.eu/resource.html?uri=cellar:e0649735-a372-11eb-9585- 
01aa75ed71a1.0001.02/DOC_1&format=PDF. 
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Conclusion 

Thank you again for this opportunity to provide initial input to the development of 
the NIST Al RMF. We are optimistic that the NIST RMF can play a key role in 
facilitating global consensus on issues related to the management of Al risks and 


help to inform related policy and regulatory efforts. We look forward to working with 
you in the months ahead to ensure that the Al RMF achieves this potential. 


Sincerely, 


JN 


Christian Troncoso 
Senior Director, Policy 


[Attachment: BSA Al Bias Framework] 
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Tremendous advances in artificial intelligence (Al) research and development are 
quickly transforming expectations about how the technology may shape the world. 
The promise that Al may one day impact every industry is quickly turning into a 
commercial reality. From financial services to healthcare, Al is increasingly leveraged 
to improve customer experiences, enhance competitiveness, and solve previously 
intractable problems. For instance, Al is enabling medical researchers to diagnose 
early-stage Alzheimer's Disease years before debilitating symptoms arise,* and it is 
helping ecologists analyze impossibly large datasets to better track the impact of their 
efforts to preserve critical habitat and prevent illegal elephant poaching in Malawi.” 


As used in this report, the term “artificial 
intelligence” refers to systems that use machine 
learning algorithms that can analyze large 
volumes of training data to identify correlations, 
patterns, and other metadata that can be used 
to develop a model that can make predictions or 
recommendations based on future data inputs. 
For example, developers used machine learning 
to create “Seeing Al,” an app that helps people 
who are blind or visually impaired navigate the 
world by providing auditory descriptions of 
objects in photographs.* Users of the app can use 
their smartphone to take pictures, and Seeing Al 
describes what appears in the photograph. To 
develop the computer vision model capable of 
identifying the objects in a picture, the system 
was trained using data from millions of publicly 
available images depicting common objects, such 
as trees, street signs, landscapes, and animals. 
When a user inputs a new image, Seeing Al in 
effect predicts what objects are in the photo by 
comparing it to the patterns and correlations that 
it derived from the training data. 
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The proliferation of Al across industries is also 
prompting questions about the design and 

use of the technology and what steps can be 
taken to ensure it is operating in a manner that 


accounts for any potential risks it may pose to 
the public. 


The use of advanced technologies in connection 
with high-stakes decisions presents both 
opportunities and risks. On the one hand, the 
adoption of Al by financial institutions has the 
potential to reduce discrimination and promote 
fairness by facilitating a data-driven approach to 
decision-making that is less vulnerable to human 
biases.* For instance, the use of Al can improve 
access to credit and housing to historically 
marginalized communities by enabling lenders to 
evaluate a greater array of data than is ordinarily 
accounted for in traditional credit reports. At the 
same time, researchers caution that flaws in the 
design, development, and/or deployment of Al 
systems have the potential to perpetuate (or even 
exacerbate) existing societal biases.* 
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Developing mechanisms for identifying and 
mitigating the risks of Al bias has therefore 
emerged as an area of intense focus for experts 
in industry, academia, and government. In just 
the past few years, a vast body of research has 
identified a range of organizational best practices, 
governance safeguards, and technical tools that 
can help manage the risks of bias throughout 
the Al lifecycle. Static evaluations of Al models 
cannot account for all potential issues that may 
arise when Al systems are deployed in the field, 
so experts agree that mitigating risks of Al 

bias requires a lifecycle approach that includes 
ongoing monitoring by end-users to ensure that 
the system is operating as intended. 










lifecycle. Similar to impact assessments for data 
privacy, Al impact assessments can serve as an 
important assurance mechanism that promotes 


accountability and enhances trust that high-risk 

Al systems have been designed, developed, 
tested, and deployed with sufficient protections 

in place to mitigate the risk of harm. Al impact 
assessments are also an important transparency 
mechanism that enables the many potential 
stakeholders involved in the design, development, 
and deployment of an Al system to communicate 
about its risks and ensure that responsibilities for 
mitigating those risks are clearly understood. 


In addition to setting forth a process for 
performing an Al impact assessment, the 
Bias Risk Management Framework: 


e Sets out the key corporate governance 
structures, processes, and safeguards that are 
needed to implement and support an effective 
Al risk management program; and 


e Identifies existing best practices, technical 
tools, and resources that stakeholders can 
use to mitigate specific Al bias risks that can 
emerge throughout an Al system's lifecycle. 


This Framework is intended to be a flexible tool that organizations 
can use to enhance trust in their Al systems through risk management 


processes that promote fairness, transparency, and accountability. 
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References to “Al bias” in this document refer to Al systems that systematically and 
unjustifiably yield less favorable, unfair, or harmful outcomes to members of specific 


demographic groups. 


At its core, the goal of machine learning is to 
create a model that derives generalized rules from 
historical examples in order to make predictions 
about future data inputs. For instance, an image 
recognition system designed to identify plants 
would likely be trained on large volumes of 
photographs depicting each of the many species 
of vegetation. The system would look for general 
rules, like leaf patterns, that are common across 
the photographs of each species, thereby 
creating a model that can evaluate whether 

new data inputs (i.e., user-submitted photos) 
include any of the species it has been trained 

to identify. In other words, machine learning 
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works by drawing generalizations from past data 
to make predictions about future data inputs. 
However, when Al is used to model human 
behavior, concerns about unintended bias take 
on an entirely different dimension. As Al is 
integrated into business processes that can have 
consequential impacts on people's lives, there 

is a risk that “biased” systems will systematically 
disadvantage members of historically marginalized 
communities. Al bias can manifest in systems 
that perform less accurately or treat people less 
favorably based on a sensitive characteristic, 
including but not limited to race, gender identity, 
sexual orientation, age, religion, or disability. 
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Sources and Types of Al Bias 


DESIGN 


Al bias can be introduced at multiple stages in the Al lifecycle.? Decisions made at the earliest stages of 
the conception and design of an Al system can introduce bias: 


e Problem Formulation Bias. In some instances, the basic assumptions underlying a proposed 
Al system may be so inherently biased that they render it inappropriate for any form of public 
deployment. 


EXAMPLES 


In 2016, researchers at Shanghai Jiao Tong University published a highly controversial paper’ 
detailing their effort to train an Al system to predict “criminality” through a facial imaging 
system. By training the system on a large volume of police mugshots, the researchers alleged 
that their system could predict “criminality” with close to 90 percent accuracy merely by 
analyzing a person’s facial structure. Unsurprisingly, the paper quickly became the subject of 
scathing criticism, and commentators rightfully noted that the model relied on the profoundly 
disturbing (and causally unsupportable) assumption that criminality can be inferred from a 
person's appearance." 


Problem formulation bias can also arise when an Al system's target variable is an imprecise 

or overly simplistic proxy for what the system is actually trying to predict. For example, in 
2019 researchers discovered that an Al system widely used by hospitals to triage patients’ 

by predicting the likelihood that they required urgent care systematically prioritized the 
needs of healthier white patients to the detriment of less-healthy minority patients. In this 
instance, bias arose because the system sought to predict “healthcare needs” using historical 
data about “healthcare costs” as an easy-to-obtain stand-in for the actual data about the 
healthcare needs of patients. Unfortunately, because minority patients have historically had 
less access to healthcare, using “healthcare costs” as a proxy for the current needs of those 
patients paints an inaccurate picture that can result in dangerously biased outcomes. 
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e Historical Bias. There is a risk of perpetuating historical biases reflected in data used to train an Al 
system. 


EXAMPLE 


A medical school in the United Kingdom set out to create a system that would help identify 
good candidates for admission. The system was trained using data about previously admitted 
students. It was discovered, however, that the school’s historical admissions decisions had 
systematically disfavored racial minorities and females whose credentials were otherwise equal to 
other applicants. By training the model using data reflecting historical biases, the medical school 
inadvertently created a system that replicated those same biased admission patterns.'® 


e Sampling Bias. If the data used to train a system is misrepresentative of the population in which 
it will be used, there is a risk that the system will perform less effectively on communities that may 
have been underrepresented in the training data. This commonly occurs when sufficient quantities 
of representative data are not readily available, or when data is selected or collected in ways that 
systematically over- or under-represent certain populations. 


EXAMPLES 


As the pathbreaking research by Joy Buolamwini and Timnit Gebru demonstrated, facial 
recognition systems trained on datasets composed disproportionately of white and male 
faces perform substantially less accurately when evaluating the faces of women with darker 
complexions." 


Sampling bias can also arise as a result of data collection practices. The City of Boston's 
attempt to create a system capable of automatically detecting and reporting potholes in 

need of repair is an illustrative case in point. Because early versions of the program relied 
heavily on data supplied by users of a smartphone app called “StreetBump,” it received a 
disproportionate number of reports from affluent neighborhoods with residents who could 
afford smartphones and data plans. As a result of the sampling bias, potholes in poorer 
neighborhoods were underrepresented in the dataset, creating a risk that the system would 
allocate repair resources in a manner that would treat members of those communities unfairly. 1? 
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e Labeling Bias. Many Al systems require training data to be “labeled” so that the learning algorithm 
can identify patterns and correlations that can be used to classify future data inputs. The process of 
labeling the training dataset can involve subjective decisions that can be a vector for introducing 
human biases into the Al system. 


EXAMPLE 


ImageNet is a database of more than 14 million images that have been categorized and 
labeled to enable Al researchers to train vision recognition systems. Although ImageNet 

has been a critical tool for advancing the state of the art in Al object recognition, recent 
scholarship has shone a light on how the database's categorization and labeling system can 
create significant risks of bias when it is used to train systems involving images of people. 

In Excavating Al,'3 Kate Crawford and Trevor Paglen demonstrated that the categories and 
data labels associated with the images of people in ImageNet reflect a range of “gendered, 
racialized, ableist, and ageist” biases that could be propagated in any Al system that uses 
them as training data. For instance, an Al system trained on ImageNet data was more likely to 
classify images of Black subjects as “wrongdoers” or “offenders.” 14 


El DEVELOPMENT 


Once the necessary data has been collected, the development team must clean, process, and normalize 
the data so that it can be used to train and validate a model. Developers must also select a machine 
learning approach, or adapt an off-the-shelf model, that is appropriate for the nature of the data they are 
using and the problem they are trying to solve. This may involve building many different models using 
different approaches and then choosing the most successful among them.** Usually, the development 
team must also make choices about data parameters to make the model functional. For instance, data 
reflecting a numerical score may be converted to a “yes” or “no” answer by assigning a threshold—for 
example, scores equal or greater to X may be re-designated as a “yes,” and scores below that threshold 
designated “no.” Biases that can emerge during the development stage include the following: 





e Proxy Bias. The process of selecting the input variables (i.e., “features”) that the model will weigh 
as it is being trained is another critical decision point that can introduce bias. Even when sensitive 
demographic data is excluded, bias may be introduced if the system relies on features that are closely 
correlated to those traits, called proxies. 


EXAMPLE 


Even the use of seemingly benign features can introduce proxy bias due to their correlation 
with sensitive attributes. Researchers have shown, for instance, that information about whether 
a person owns a Mac or PC laptop may be predictive of their likelihood to pay back a loan.'® A 
financial institution might therefore seek to include such a variable when building an Al system 
to screen potential loan applicants. However, the inclusion of that feature also introduces a 
significant risk of proxy bias because Mac ownership correlates closely to race. As a result, its 
inclusion could result in a system that systematically disfavors applicants based on a feature that 
is closely correlated to race but that is unrelated to actual credit risk. 
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e Aggregation Bias. Using a “one-size-fits-all” model that overlooks key variables can result in system 
performance that is optimized only for the dominant sub-group. Aggregation bias can arise if the 
model fails to account for underlying differences between sub-groups that materially impact a 
system's accuracy rates. Rare phenomena may be lost in averages and aggregates. Worse, models of 
aggregated populations may correctly predict different or even opposite behavior to modes of sub- 
groups of the same population, a phenomenon known as Simpson's Paradox. 


EXAMPLE 


The risk of aggregation bias is particularly acute in healthcare settings where diagnosis and 
treatment must often account for the unique manner in which medical conditions may impact 
people across racial and ethnic lines. For instance, because the risk of complications posed by 
diabetes varies wildly across ethnicities, an Al system used to predict the risks associated with 
diabetes may underperform for certain patients unless it accounts for these differences." 





=i DEPLOYMENT, MONITORING, AND ITERATION 


Al systems inevitably encounter real world scenarios that differ from the data used to train the model. 
As a result, even a system that has been thoroughly validated and tested prior to deployment may 
suffer performance degradation when it is put into production. Therefore, it is important that Al systems 
undergo ongoing evaluation and assessment throughout their lifecycles. 


e Deployment Bias. Bias can arise in various ways after a system has been deployed, including when 
the data used to train or evaluate an Al system differs markedly from the population the system 
encounters when it is deployed, rendering the model unable to perform as intended. Deployment 
bias can emerge when a model is unable to reliably generalize beyond the data on which it was 
trained, either because the model was overfitted at the time of training (i.e., the prediction model 
learned so much detail about the training data that it is unable to make accurate generalizations 
about other data inputs) or because of concept drift (i.e., performance degradation was brought on 
by a shift in the relationship between the target variable and the training data). 


e Misuse Bias. Deployment bias can also arise when an Al system or feature built for one purpose is 
used in an unexpected or unintended manner. 
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What Is Risk Management? 


Risk management is a process for ensuring systems are trustworthy by design by 
establishing a methodology for identifying risks and mitigating their potential 
impact. Risk management processes are particularly important in contexts, such as 
cybersecurity and privacy, where the combination of quickly evolving technologies 
and highly dynamic threat landscapes render traditional “compliance” based 
approaches ineffective. Rather than evaluating a product or service against a static 
set of prescriptive requirements that quickly become outdated, risk management 
seeks to integrate compliance responsibilities into the development pipeline to help 

mitigate risks throughout a product or services lifecycle. Effective risk management is 
anchored around a governance framework that promotes collaboration between an 


organization's development team and its compliance personnel at key points during 
the design, development, and deployment of a product. 
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Managing the Risk of Bias 


Organizations that develop and use Al systems must take steps to prevent bias from manifesting in a 
manner that unjustifiably yields less favorable or harmful outcomes based on someone's demographic 
characteristics. Effectively guarding against the harms that might arise from such bias requires a risk 
management approach because: 
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Foundations Tor Effective 


Risk Management . 


The aim of risk management is to establish repeatable processes for identifying 
and mitigating potential risks that can arise throughout an Al system’s lifecycle. A 
comprehensive risk management program has two key elements: 





BSA | The Softwaré Alliance ` 





Confronting Bias: BSA's Framework to Build Trust in Al 


Governance Framework 


Effective Al risk management should be underpinned by a governance framework that establishes the 
policies, processes, and personnel that will be used to identify, mitigate, and document risks throughout 
the system's lifecycle. The purpose of such a governance framework is to promote understanding 

across organizational units—including product development, compliance, marketing, sales, and senior 
management—about each entity's role and responsibilities for promoting effective risk management 
during the design, development, and deployment of Al systems. Key features of a risk management 
governance framework include: 


Policies and Processes 


At the core of the governance framework is a set of formal policies setting forth the organization's 
approach to risk management. These policies should define the organization’s risk management 
objectives, the procedures that it will use to meet those objectives, and the benchmarks it will rely 
on for evaluating compliance. 


e Objectives. Al risk management should be contextualized within an organization's broader risk 
management functions with the goal of ensuring that the organization is developing and using Al in 
a manner that aligns with its core values. To that end, the governance framework should identify how 
the organization will manage risks that could undermine those values. 


e Processes. The governance framework should establish processes and procedures for identifying 
risks, assessing the materiality of those risks, and mitigating risks at each stage of the Al lifecycle. 


e Evaluation Mechanisms. The governance framework should establish mechanisms, such as metrics 
and benchmarks, that the organization will use to evaluate whether policies and procedures are being 
carried out as specified. 


e Periodic Review. As Al capabilities continue to mature and the technology is put to new uses, it is 
important that organizations periodically review and update their Al governance framework so that 
it remains fit-for-purpose and capable of addressing the evolving landscape of risk. 


Bla 8 

a sH Executive Oversight. Al Developers and Al Deployers should maintain a governance 

aja framework that is backed by sufficient executive oversight. In addition to developing and 
approving the substance of the governance framework's policies, senior management 
should play an active role in overseeing the company’s Al product development lifecycle. 
For high-risk systems that may negatively impact people in consequential ways, company 
leadership should be accountable for making “go/no-go” decisions. 
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Personnel, Roles, and Responsibilities 


The effectiveness of risk management depends on establishing a cross-functional group of experts that 
can guide decisions throughout the Al lifecycle. Depending on the size of an organization and the nature 
of the systems it is developing or deploying, the responsibilities for risk management may involve staff 
from multiple business units. The governance framework should therefore identify the personnel within 
the organization who have roles and responsibilities related to Al risk management and clearly map 
reporting lines, authorities, and necessary expertise. In assigning roles and responsibilities, organizations 
should prioritize independence, competence, influence, and diversity. 


e Independence. Risk management is most effective when personnel are structured in a manner that 
facilitates separate layers of independent review. For instance, risk management responsibilities may 
be split between multiple teams, including: 


— Product Development Team. Engineers, data scientists, and domain experts involved in 
designing and developing Al products and services. 


— Compliance Team. A diverse team of legal, compliance, domain experts, and data professionals 
who are responsible for overseeing compliance with the company's Al development policies and 
practices, such as the development of impact assessments for high-risk Al systems. 


— Governance Team. Ideally a senior management-led team with responsibility for developing, 
maintaining, and ensuring effective oversight of the organization's Al Governance Framework and 
risk management processes. 


e Competence, Resourcing, and Influence. Personnel with risk management responsibilities must 
be provided with adequate training and resources to fulfill their governance functions. It is equally 
important to ensure that personnel are empowered and have the right incentives to make decisions 
to address and/or escalate risks. For instance, the organization should establish a clear escalation path 
that enables risk management personnel to engage with executive decision-makers so that there is 
executive-level visibility into key risk areas and decisions. 


We 


© Diversity. The sociotechnical nature of Al systems makes it vitally important to 

ÉY È prioritize diversity within the teams involved in a system's development and oversight. 
Development and oversight processes are most effective when team members bring 
diverse perspectives and backgrounds that can help anticipate the needs and concerns 
of users who may be impacted by or interact with an Al system. Because “algorithm 
development implicitly encodes developer assumptions that they may not be aware 
of, including ethical and political values,” it is vital that organizations establish teams 
that reflect a diversity of lived experiences and that traditionally underrepresented 
perspectives are included throughout the lifecycle of the Al design and development 
process.”° To the extent an organization is lacking in diversity, it should consult with 
outside stakeholders to solicit feedback, particularly from underrepresented groups that 
may be impacted by the system. 
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Impact Assessment 


To effectively manage Al risks, organizations should implement a robust process for performing impact 
assessments on any system that may materially impact members of the public. Impact assessments 

are widely used in a range of other fields—from environmental protection to data protection—as an 
accountability mechanism that promotes trust by demonstrating that a system has been designed in a 
manner that accounts for the potential risks it may pose to the public. In short, the purpose of an impact 
assessment is to identify the risks that a system may pose, quantify the degree of harm the system could 
generate, and document any steps that have been taken to mitigate those risks to an acceptable level. 


Impact assessment processes should be tailored to address the nature of the system that is being 
evaluated and the type of harms it may pose. For truly low-risk systems—for example, a system used to 
predict the type of fonts being used on a document—a full impact assessment may not be necessary. 
But for systems that pose an inherent risk of material harm to the public, a full impact assessment should 
be performed. Given the incredible range of applications to which Al can be applied, there is no “one- 
size-fits-all” approach for identifying and mitigating risks. Instead, impact assessment processes should 
be tailored to address the nature of an Al system and the type of inherent risks and potential harms it 
may pose. To determine whether a system poses an inherent risk of material harm, stakeholders should 
consider: 


e Potential Impact on People. Impact assessments are likewise important in circumstances where an Al 
system will be used in decision-making processes that may result in consequential impacts on people, 
such as their ability to obtain access to credit or housing. 


e Context and Purpose of the System. Evaluating the nature of the Al system and the setting in which 
it will be used is a good starting point for determining both the necessity and appropriate scope of 
an impact assessment. Impact assessments are particularly critical for high-risk Al systems that will 
be used in domains (e.g., healthcare, transportation, finance) where the severity and/or likelihood of 
potential harms is high. 


e Degree of Human Oversight. The degree to which an Al system is fully automated may also impact 
the inherent risks that it poses. A system designed to provide recommendations to a highly skilled 
professional is likely to pose fewer inherent risks than a similarly situated fully automated system. Of 
course, the mere existence of a human-in-the-loop certainly does not mean that an Al system is free 
from risk. It is necessary instead to examine the nature of the human-computer interaction holistically 
to determine the extent to which human oversight may mitigate an Al system's inherent risks. 


e Type of Data. The nature of the data used to train a system can also shed light on a system's inherent 
risks. For instance, using training data relating to human characteristics or behaviors is a signal that a 
system may require closer scrutiny for bias. 
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We outline below an Al Bias Risk Management Framework that is intended to aid 
organizations in performing impact assessments on systems with potential risks of Al 
bias. In addition to setting forth processes for identifying the sources of bias that can 
arise throughout an Al system’s lifecycle, the Framework identifies best practices that 


can be used to mitigate those risks. 


The Framework is an assurance-based accountability mechanism that can be used 
by Al Developer and Al Deployer organizations for purposes of: 


e Internal Process Guidance. Al Developers e Vendor Relations. Al Deployers may choose 
and Al Deployers can use the Framework as to use the Framework to guide purchasing 
a tool for organizing and establishing roles, decisions and/or developing vendor contracts 
responsibilities, and expectations for internal that ensure Al risks have been adequately 
processes. accounted for. 

e Training, Awareness, and Education. e Trust and Confidence. Al Developers may 
Al Developers and Al Deployers can use wish to communicate information about 
the Framework to build internal training and a product's features and its approach to 
education programs for employees involved in mitigating Al bias risks to a public audience. 
developing and using Al systems. In addition, In that sense, the Framework can help 
the Framework may provide a useful tool for organizations communicate to the public 
educating executives about the organization 's about their commitment to building ethical 
approach to managing Al bias risks. Al systems. 

e Assurance and Accountability. Al e Incident Response. Following an unexpected 
Developers and Al Deployers can use the incident, the processes and documentation 
Framework as a basis for communicating set forth in the Framework provide an audit 
and coordinating about their respective roles trail that can help Al Developers and Al 
and responsibilities for managing Al risks Deployers identify the potential source of 
throughout a system's lifecycle. system underperformance or failure. 
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Al Lifecycle Phases 


The Framework is organized around the phases of the Al lifecycle, which represent the key iterative steps 
involved in the creation and use of an Al system. 


7 


DESIGN PHASE 





e Project Conception. The initial stage of Al design involves identifying and formulating the “problem” 
that the system is intended to address and initially mapping how the model will achieve that 
objective. During this phase, the design team will define the purpose and structure of the system. 
Depending on the nature of the system, the design team will identify a target variable that the 
system is intended to predict. For instance, a fitness app that analyzes a consumer's heart rate to 
monitor for irregularities that might predict whether that person is at risk of a stroke or heart disease 
(i.e., the target variable). At this early stage of the system design process, the goal of the Bias Risk 
Management Framework is to identify whether using Al is appropriate for the project at hand. 
Potential risks include: 


— Problem Formulation Bias. Target variables may reflect inherent prejudices or faulty assumptions 
that can perpetuate harmful biases. In some instances, the basic assumptions underlying a 
proposed Al system may be so inherently biased as to render it inappropriate for any form of 
public deployment. 


e Data Acquisition. Once the system objectives have been defined, developers must assemble a 
corpus of data that will be used to train the model to identify patterns that will enable it to make 
predictions about future data inputs. This training data can inadvertently introduce biases into an 
Al system in many ways. Potential risks include: 


— Historical Bias. Training an Al system using data that itself may reflect historical biases creates a 
risk of further entrenching those inequities. 


— Sampling Bias. The risk of bias also arises when the data used to train an Al system is 
not representative of the population in which it will be deployed. An Al system trained on 
unrepresentative data may not operate as effectively when making predictions about a 
member of a class that is either over- or under-represented. 


— Labeling Bias. Many Al systems require training data to be labeled so that it can identify what 
patterns it should be looking for. The process of labeling the training dataset can be a vector for 
introducing bias into the Al system. 


www.bsa.org 15 


Confronting Bias: BSA's Framework to Build Trust in Al 


16 


DEVELOPMENT PHASE 


e Data Preparation and Model Definition. The next step of the Al lifecycle involves preparing the data 


so that it is ready to train the model. During this process, the development team will clean, normalize, 
and identify the variables (i.e., “features”) in the training data that the algorithm will evaluate as it 
looks for patterns and relationships as the basis of a rule for making future predictions. The team must 
also establish the system's underlying architecture, including selecting the type of algorithmic model 
that will power the system (e.g., linear regression, logistic regression, deep neural network.)? Once 
the data is ready and the algorithm is selected, the team will train the system to produce a functional 
model that can make predictions about future data inputs. Potential risks include the following: 


— Proxy Bias. The process of selecting features in the training data and choosing a modeling 
approach involves human decisions about what variables should be considered as relevant for 
making predictions about the model's target variable. These interventions can inadvertently 
introduce bias to the system, including by relying on variables that act as proxies for protected 
classes. 


— Aggregation Bias. Aggregation bias can arise if the model fails to account for underlying 
differences between sub-groups that materially impact a system's accuracy rates. Using a “one- 
size-fits-all” model that overlooks key variables can result in system performance that is optimized 
only for the dominant sub-group. 


Model Validation, Testing, and Revision. After the model has been trained, it must be validated 
to determine if it is operating as intended and tested to demonstrate that the system's outputs do 
not reflect unintended bias. Based on outcome of validation and testing, the model may need to be 
revised to mitigate risks of bias that are deemed unacceptable. 





DEPLOYMENT PHASE 





e Deployment and Use. Prior to deployment, the Al Developer should evaluate the system to 


determine whether risks identified in earlier stages of design and development have been sufficiently 
mitigated in a manner that corresponds to the company’s governance policies. To the extent 
identified risks may arise through misuse of the system, the Al Developer should seek to control for 
them by integrating product features (e.g., user interfaces that reduce risk of misuse) to mitigate 
those risks, prohibiting uses that could exacerbate risks (e.g., end-user license agreements), and 
providing Al Deployers with sufficient documentation to perform their own impact assessments. 


Prior to using an Al system, an Al Deployer should review documentation provided by the Al 
Developer to assess whether the system corresponds with its own Al governance policies and to 
determine whether deployment-related risk management responsibilities are clearly assigned. 
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Although some post-deployment risk management responsibilities may be addressed by the Al 
Developer, the Al Deployer will often bear responsibility for monitoring system performance and 
evaluating whether it is operating in a manner that is consistent with its risk profile. Potential risks 


include: 


— Deployment Bias. Al systems are trained on data that represents a static moment in time and 


that filters out “noise” that could undermine the model's ability to make consistent and accurate 
predictions. Upon deployment in the real world, Al systems will necessarily encounter conditions 
that differ from those in the development and testing environment. Further, because the real- 
world changes over time, the snapshot in time that a model represents may naturally become less 
accurate as the relationship between data variables evolves. If the input data for a deployed Al 
system differs materially from its training data, there is a risk that the system could “drift” and that 
the performance of the model could be undermined in ways that will exacerbate the risks of bias. 
For instance, if an Al system is designed (and tested) for use in a specific country, the system may 
not perform well if it is deployed in a country with radically different demographics. 


Misuse Bias. Deploying an Al system into an environment that differs significantly from the 
conditions for which it was designed or for purposes that are inconsistent with its intended use 


cases can exacerbate risks of bias. 


Framework Structure 


The Framework identifies best practices for identifying and mitigating risks of Al bias across the entire 
system lifecycle. It is organized into: 


Functions, which denote fundamental Al risk 
management activities at their highest level, 
dividing them between Impact Assessment and 
Risk Mitigation Best Practices. 


Categories, which set out the activities and 
processes that are needed to execute upon 
the Functions at each phase of the Al Lifecycle. 
In other words, the Categories set forth the 
steps for performing an Impact Assessment 
and identify the corresponding Risk Mitigation 
Best Practices that can be used to manage 
associated risks. 


Diagnostic Statements, which set forth 

the discrete actions that should be taken to 
execute upon the Categories. They provide a 
set of results that help support achievement of 
the outcomes in each Category. 
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Comments on Implementation, which 
provide additional information for achieving 
the outcomes described in the Diagnostic 
Statements. 


Tools and Resources, which identify a 

range of external guidance and toolkits that 
stakeholders can use to mitigate the bias risks 
associated with each phase of the Al lifecycle. 
The specific tools and resources identified 

in the framework are non-exhaustive and are 
highlighted for informational purposes only. 
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Stakeholder Roles and 
Responsibilities 


Reflecting the inherently dynamic nature of Al 
systems, the Framework is intended to account 
for the array of stakeholders that may play a 

role in various aspects of a system's design, 
development, and deployment. Because 

there is no single model of Al development or 
deployment, it is impossible in the abstract to 
assign roles or delegate specific responsibilities 
for many of the Framework's risk management 
functions. However, in general, there are three 
sets of stakeholders that may bear varying 
degrees of responsibility for certain aspects of Al 
risk management throughout a system’s lifecycle: 


e Al Developers. Al Developers are 
organizations responsible for the design and 
development of Al systems. 


e Al Deployers. Al Deployers are the 
organizations that adopt and use Al systems. (If 
an entity develops its own system, it is both the 
Al Developer and the Al Deployer.) 


e Al End-Users. Al End-Users are the 
individuals—oftentimes an employee of an Al 
Deployer—who are responsible for overseeing 
the use of an Al system. 


The allocation of risk management responsibilities 
between these stakeholders will in many cases 
depend on an Al system's development and 
deployment model. 


Spectrum of Al Development 
and Deployment Models 


The appropriate allocation of risk management 
responsibilities between stakeholders will vary 
depending on the nature of the Al system being 
developed and which party determines the 
purposes and means by which the underlying 
model is trained. For instance: 


e Universal, Static Model. The Al Developer 
provides all its customers (i.e., Al Deployers) 
with a static, pre-trained model. 


— The Al Developer will bear responsibility for 
most aspects of model risk management. 


e Customizable Model. The Al Developer 
provides a pre-trained model to Al Deployers 
who can customize and/or retrain the model 
using their own data. 


— Risk management will be a shared 
responsibility between the Al Developer 
and the Al Deployer. 


e Bespoke Model. The Al Developer trains a 
bespoke Al model on behalf of an Al Deployer 
using the Al Deployer's data. 


— Risk management will be a shared 
responsibility between the Al Developer 
and the Al Deployer, with the bulk of 
obligations falling on the Al Deployer. 
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| Function | Category Diagnostic Statement Comments on Implementation 





Impact 
Assessment 


Identify and 
Document Objectives 
and Assumptions 


Select and Document 
Metrics for Evaluating 
Fairness 


Document 
Stakeholder Impacts 


Document Risk 
Mitigations 
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Document the intent and purpose of the 
system. 


Clearly define the model's intended 
effects. 


Clearly define intended use cases and 
context in which the system will be 
deployed. 


Identify “fairness” metrics that will be 
used as a baseline for assessing bias in 
the Al system. 


Identify stakeholder groups that may be 
impacted by the system. 


For each stakeholder group, document 
the potential benefits and potential 
adverse impacts, considering both 

the intended uses and reasonably 
foreseeable misuses of the system. 


Assess whether the nature of the system 
makes it prone to potential bias-related 
harms based on user demographics. 


If risk of bias is present, document efforts 
to mitigate risks. 


e What is the purpose of the system—i.e., 
what “problem” will it solve? 


e Who is the intended user of the system? 
e Where and how will the system be used? 


e What are the potential misuses? 


What is the model intended to predict, 
classify, recommend, rank, or discover? 


The concept of “fairness” is highly subjective 
and there are dozens of metrics by which it 
can be evaluated. Because it is impossible to 
simultaneously satisfy all fairness metrics, it 
is necessary to select metrics that are most 
appropriate for the nature of the Al system 
that is being developed and consistent 

with any applicable legal requirements. It 

is important to document the rationale by 
which fairness metrics were selected and/ 

or excluded to inform latter stages of the Al 
lifecycle. 


Stakeholder groups include Al Deployers, 

Al End-Users, Affected Individuals (i.e., 
members of the public who may interact with 
or be impacted by an Al system). 


User demographics may include, but are not 
limited to race, gender, age, disability status, 
and their intersections. 
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| Function Category Diagnostic Statement Comments on Implementation 











Document how identified risks and 
potential harms of each risk will be 
measured and how the effectiveness of 
mitigation strategies will be evaluated. 


Document Risk 
Mitigations 


Impact 
Assessment 
(continued) 


If risk of bias is present, document efforts 
to mitigate risks. 


If risks are unmitigated, document why 
the risk was deemed acceptable. 


Seek feedback from a diverse set of 
stakeholders to inform the impact 
assessment. 


Because risks identified during this initial 
phase will inform later aspects of the 
development and impact assessment 
processes, it is vital to develop a holistic 
understanding of potential harms that may 
arise by soliciting diverse perspectives from 
people with a range of lived experiences, 
cultural backgrounds, and subject matter 
expertise. To the extent in-house personnel 
lack subject matter or cultural diversity, it 
may be necessary to consult with third-party 
experts or to solicit feedback from members 
of communities that may be adversely 
impacted by the system. 


Risk Mitigation 
Best Practices 


Independence and 
Diversity 


Share impact assessment documentation 
with personnel working on later stages of 
the Al pipeline so that risks and potential 
unintended impacts can be monitored 
throughout the development process. 


Transparent 
Documentation 


Accountability and 
Governance 


Ensure that senior leadership has been 
adquately briefed on potential high risk 
Al systems. 


Impact assessment documentation for 
systems deemed “high risk” should be 
shared with senior leadership to facilitate a 
“go/no-go" decision. 


Maintain Records of 
Data Provenance 


Impact 
Assessment 
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Maintain sufficient records to enable 
“recreation” ofthe data used to train 
the Al model, verify that its results are 
reproducible, and monitor for material 
updates to data sources. 


Records should include: 

e Source of data 

e Origin of data (e.g., Who created it? 
When? For what purpose? How was it 
created?) 

e Intended uses and/or restrictions of the 
data and data governance rules (e.g., What 
entity owns the data? How long can it be 
retained (or must it be destroyed)? Are 
there restrictions on its use?) 

e Known limitations of data (e.g., missing 
elements?) 

e |f data is sampled, what was the sampling 
strategy? 

e Will the data be updated? If so, will any 
versions be tracked? 
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DATA ACQUISITION 


Examine Data for 
Potential Biases 


Impact 
Assessment 
(continuea) 


Document Risk 
Mitigations 


Risk Mitigation 
Best Practices 


Independence and 
Diversity 


Re-Balancing 
Unrepresentative Data 
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Scrutinize data for historical biases. 


Evaluate “representativeness” of the 
data. 


Scrutinize data labeling methodology. 


Document whether and how data was 
augmented, manipulated, or re-balanced 
to mitigate bias. 


To facilitate robust interrogation of the 
datasets, data review teams should 
include personnel that are diverse in 
terms of their subject matter expertise 
and lived experiences. 


Consider re-balancing with additional 
data. 


Consider re-balancing with synthetic data. 


| Function Category Diagnostic Statement Comments on Implementation 


Examine sources of data and assess potential 
that they may reflect historical biases. 


e Compare demographic distribution of 
training data to the population where the 
system will be deployed. 


e Assess whether there is sufficient 
representation of subpopulations that are 
likely to interact with the system. 


e Document personnel and processes used 
to label data. 


e For third-party data, scrutinize labeling 
(and associated methodologies) for 
potential sources of bias. 


Effectively identifying potential sources of 
bias in data requires a diverse set of expertise 
and experiences, including familiarity with 
the domain from which data is drawn and a 
deep understanding of the historical context 
and institutions that produced it. To the 
extent in-house personnel lack diversity, 
consultation with third-party experts or 
potentially affected stakeholder groups may 
be necessary. 


Improving representativeness can be 
achieved in some circumstances by collecting 
additional data that improves the balance of 
the overall training dataset. 


Imbalanced datasets can potentially be 
rebalanced by “oversampling” data from 
the underrepresented groups. A common 
oversampling method is the Synthetic 
Minority Oversampling Technique, which 
generates new “synthesized” data from the 
underrepresented group. 
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E) DESIGN 
e 


DATA ACQUISITION 





Risk Mitigation Data Labeling Establish objective and scalable labeling e To mitigate the potential of labeling bias, 
Best Practices guidelines. the personnel responsible for labeling 
(continued) the data should be provided with clear 


guidelines establishing an objective and 
repeatable process for individual labeling 
decisions. 


e In domains where the risk of bias is high, 
labelers should have adequate subject 
matter expertise and be provided training 
to recognize potential unconscious biases. 


e For high-risk systems, it may be necessary 
to set up a quality assurance mechanism to 
monitor label quality. 


Accountability and Integrate data labeling processes into a Establishing an organizational data strategy 
Governance comprehensive data strategy. can help ensure that data evaluation 
is performed consistently and prevent 
duplication of effort by ensuring that 
company efforts to scrutinize data are 
documented for future reference. 


DESIGN: RISK MITIGATION TOOLS AND RESOURCES 





Project Conception Data Compilation 
e Aequitas Bias and Fairness Audit Toolkit e Datasheets for Datasets 
Pedro Saleiro, Abby Stevens, Ari Anisfeld, and Rayid Ghani, Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer 
University of Chicago Center for Data Science and Public Policy Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate 
(2018), htto://www.datasciencepublicpolicy.org/projects/aequitas/. Crawford, arXiv:1803.09010v7, (March 19, 2020), httos://arxiv.org/ 
e Diverse Voices Project | A How-To Guide for Facilitating abs/1803.07010. 
Inclusiveness in Tech Policy e Al FactSheets 360 
Lassana Magassa, Meg Young, and Batya Friedman, University of IBM Research, https://aif360.mybluemix.net/. 





Washington Tech Policy Lab, https://techpolicylab.uw.edu/project/ 
diverse-voices/. 
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Catego Diagnostic Statement Comments on Implementation 
gory fe] P 











Impact Document Feature Document rationale for choices made Examine whether feature selection or 
Assessment Selection and during the feature selection and engineering choices may rely on implicitly 
Engineering Processes engineering processes and evaluate their biased assumptions. 

impact on model performance. 


Document potential correlation For features that closely correlate to a 
between selected features and sensitive sensitive class, document the relevance to 
demographic attributes. the target variable and the rationale for its 


inclusion in the model. 


Document Model Document rationale for the selected 
Selection Process modeling approach. 


Identify, document, and justify 
assumptions in the selected approach 
and potential resulting limitations. 


Risk Mitigation Feature Selection Examine for biased proxy features. e Simply avoiding the use of sensitive 

Best Practices attributes as inputs to the system—an 
approach known as “fairness through 
unawareness”—is not an effective 
approach to mitigating the risk of bias. 
Even when sensitive characteristics are 
explicitly excluded from a model, other 
variables can act as proxies for those 
characteristics and introduce bias into the 
system. To avoid the risk of proxy bias, the 
Al Developer should examine the potential 
correlation between a model's features 
and protected traits and examine what role 
these proxy variables may be playing in the 
model's output. 


e The ability to examine statistical 
correlation between features and 
sensitive attributes may be constrained in 
circumstances where an Al Developer lacks 
access to sensitive attribute data and/ 
or is prohibited from making inferences 
about such data.” In such circumstances, a 
more holistic analysis informed by domain 
experts may be necessary. 
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| Function | Category Diagnostic Statement Comments on Implementation 





Risk Mitigation Feature Selection 
Best Practices 


(continued) 


Independence and 
Diversity 


Model Selection 


Scrutinize features that correlate to 
sensitive attributes. 


Seek feedback from diverse stakeholders 
with domain-specific expertise. 


Avoid inscrutable models in 
circumstances where both the risk and 
potential impact of bias are high. 


e Features that are known to correlate to a 
sensitive attribute should only be used if 
there is a strong logical relationship to the 
system's target variable. 


e For example, income—although correlated 
to gender—is reasonably related to a 
person's ability to pay back a loan. The 
use of income in an Al system designed to 
evaluate creditworthiness would therefore 
be justified. In contrast, the use of “shoe 
size"—which also correlates to gender—in 
a model for predicting creditworthiness 
would be an inappropriate use of a 
variable that closely correlates to a 
sensitive characteristic. 


The feature engineering process should 

be informed by personnel with diverse 

lived experiences and expertise about the 
historical, legal, and social dimensions of the 
data being used to train the system. 


Using more interpretable models can 
mitigate the risks of unintended bias by 
making it easier to identify and mitigate 
problems. 


Document Validation 
Processes 


Impact 
Assessment 


Document Testing 
Processes 
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Document how the system (and individual 
components) will be validated to evaluate 
whether it is performing consistent with 
the design objectives and intended 
deployment scenarios. 


Document re-validation processes. 


Test the system for bias by evaluating and 
documenting model performance. 


Document how testing was performed, 
which fairness metrics were evaluated, 
and why those measures were selected. 


Document model interventions. 


e Establish cadence at which model will be 
regularly re-validated. 


e Establish performance benchmarks that 
will trigger out-of-cycle re-validation. 


Testing should incorporate fairness metrics 
identified during Design phase and examine 
the model's accuracy and error rates across 
demographic groups. 


If testing reveals unacceptable levels of bias, 
document efforts to refine the model. 


BSA | The Software Alliance 


IG DEVELOPMENT 


Confronting Bias: BSA's Framework to Build Trust in Al 








VALIDATING, TESTING, AND REVISING THE MODEL 


Risk Mitigation Model Interventions 


Best Practices 


Independence and 


development. 


DEVELOPMENT: RISK MITIGATION TOOLS AND RESOURCES 


e Model Cards for Model Reporting 


Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy 


Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, 
and Timnit Gebru, Proceedings of the 2019 Conference on Fairness, 
Accountability, and Transparency, (January 2019): 220-229, https:// 
arxiv.org/abs/1810.03993. 


e Al Factsheets 360 
Aleksandra Mojsilovic, IBM Research (August 22, 2018), 
https://www.ibm.com/blogs/research/2018/08/factsheets-ai/. 


e Al Explainability 360 
IBM Research, https://aix360.mybluemix.net/. 


e Al Fairness 360 
IBM Research, https://aif360.mybluemix.net/. 














e Responsible Machine Learning with Error Analysis 
Besmira Nushi, Microsoft Research (February 18, 2021), 
https://techcommunity.microsoft.com/t5/azure-ai/responsible- 
machine-learning-with-error-analysis/ba-p/2141774. 








www.bsa.org 


Evaluate potential model refinements to 
address bias surfaced during testing. 


Validation and testing documentation 
Diversity should be reviewed by personnel 
who were not involved in the system's 


| Function | Category Diagnostic Statement Comments on Implementation 


In circumstances where testing reveals that 
the system is exhibiting unacceptable levels 
of bias based on the selected fairness metric, 
it will be necessary to refine the model. 
Potential model refinements include: 


e Pre-Processing Interventions. Such 
refinements can involve revisiting earlier 
stages of the Design and Development 
lifecycle (e.g., seeking out additional 
training data). 


e In-Processing Interventions. Bias can also 
be mitigated by imposing an additional 
fairness constraint directly on the model. 
Traditional machine learning models 
are designed to maximize for predictive 
accuracy. Emerging techniques enable 
developers to build constraints into the 
model to reduce the potential for bias 
across groups. The addition of a fairness 
constraint, in effect, instructs the model to 
optimize both for accuracy and a specific 
fairness metric. 


e Post-Processing Interventions. In some 
cases, bias can be addressed through the 
use of post-processing algorithms that 
manipulate the model's output predictions 
to ensure that it adheres to a desired 
distribution. 


The independent team should compare the 
validation and testing results to the system 
specifications developed during earlier 
phases of the design and development 
process. 


e Aequitas Open Source Bias Audit Toolkit 


Pedro Saleiro, Abby Stevens, Ari Anisfeld, and Rayid Ghani, University 
of Chicago Center for Data Science and Public Policy, http://www. 
datasciencepublicpolicy.org/projects/aequitas/. 





FairTest: Discovering Unwarranted Associations in Data-Driven 
Applications 

Florian Tramer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, 
Jean-Pierre Hubaux, Mathias Humbert, Ari Juels and Huang Lin, 
ArXiv, (2015), httos://github.com/columbia/fairtest. 


Bayesian Improved Surname Geocoding 
Consumer Finance Protection Bureau (2014), https://files. 
consumerfinance.gov/f/201409 cfpb report proxy-methodology.pdf. 
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DEPLOYMENT AND USE 


Category Diagnostic Statement Comments on Implementation 





Document Lines of 
Responsibility 


Impact 
Assessment 


Document Processes 
for Monitoring Data 


Document Processes 
for Monitoring Model 
Performance 


Document Audit and 
End-of-Life Processes 


Risk Mitigation 
Best Practices 


Monitoring for 
Drift and Model 
Degradation 
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Define and document who is responsible 
for the system's outputs and the 
outcomes they may lead to, including 
details about how a system's decisions 
can be reviewed if necessary. 


Establish management plans for 
responding to potential incidents or 
reports of system errors. 


Document what processes and metrics 
will be used to evaluate whether 
production data (i.e., input data the 
system encounters during deployment) 
differs materially from training data. 


For static models, document how 
performance levels and classes of 
error will be monitored over time and 
benchmarks that will trigger review. 


For models that are intended to evolve 
over time, document how changes will be 
inventoried; if, when, and how versions 
will be captured and managed; and how 
performance levels will be monitored 
(e.g., cadence of scheduled reviews, 
performance indicators that may trigger 
out-of-cycle review). 


Document the cadence at which impact 
assessment evaluations will be audited to 
evaluate whether risk mitigation controls 
remain fit for purpose. 


Document expected timeline that system 
support will be provided and processes 
for decommissioning system in event that 
it falls below reasonable performance 
thresholds. 


Input data encountered during 
deployment can be evaluated against 
a statistical representation of the 
system's training data to evaluate the 
potential for data drift (i.e., material 


differences between the training data and 
deployment data that can degrade model 


performance). 


What does it mean for the system to fail 
and who might be harmed by a failure? 


How will failures be detected? 


Who will respond to failures when they are 
detected? 


Can the system be safely disabled? 


Are there appropriate plans for continuity 
of critical functions? 
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DEPLOYMENT AND USE 


Category Diagnostic Statement Comments on Implementation 


Risk Mitigation Product Features and Integrate product and user interface 
Best Practices User Interface features to mitigate risk of foreseeable 
(continuea) unintended uses—e.g., interface 





that enforces human-in-the-loop 
requirements, alerts to notify when a 
system is being misused. 


System Al Developers should provide sufficient If necessary, Al Developers can also provide 
Documentation documentation regarding system Al Deployers with a technical environment to 
capabilities, specifications, limitations, perform an independent impact assessment. 


and intended uses to enable Al 
Deployers to perform independent 
impact assessment concerning 
deployment risks. 


Consider incorporating terms into the 
End-User License Agreement that set 
forth limitations designed to prevent 
foreseeable misuses (e.g., contractual 
obligations to ensure end-user will 
comply with acceptable use policy). 


Sales and marketing materials should 
be closely reviewed to ensure that they 
are consistent with the system's actual 


capabilities. 

Al User Training Al Deployers should provide training for For human-in-the-loop oversight of Al system 
Al Users regarding a system's capabilities to be an effective risk mitigation measure, 
and limitations, and how outputs should Al Users should be provided adequate 
be evaluated and integrated into a information and training so they can 
workflow. understand how the system is operating and 


make sense of the model's outputs. 


Incident Response and Al Deployers should maintain a feedback For consequential decisions, Affected 
Feedback Mechanisms mechanism to enable Al Users and Individuals should be provided with an 
Affected Individuals (i.e., members of the appeal mechanism. 
public that may interact with the system) 
to report concerns about the operation of 
a system. 


DEPLOYMENT AND USE: RISK MITIGATION TOOLS AND RESOURCES 


e Al Incident Response Checklist e Detect Data Drift on Datasets 

BNH.AI, httes://www.bnh.ai/public-resources. Microsoft Azure Machine Learning (June 25, 2020), https://docs. 
microsoft.com/en-us/azure/machine-learning/how-to-monitor- 
datasets ?tabs=python#create-dataset-monitors. 








e Watson OpenScale 
IBM, https://www.ibm.com/cloud/watson-openscale. 
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Foundational Resources 


A Framework for Understanding Unintended 
Consequences of Machine Learning 

Harini Suresh and John V. Guttag, arXiv (February 
2020), https://arxiv.org/abs/1901.10002. 





Al Fairness 

Trisha Mahoney, Kush R. Varshney, and Michael 
Hind, O'Reilly (April 2020), https://www. oreilly. 

com/library/view/ai-fairness/9781492077664/. 








Beyond Explainability: A Practical Guide to 
Managing Risk in Machine Learning Models 
Andrew Burt, Brenda Leong, Stuart Shirrell, and 
Xiangnong (George) Wang, Future of Privacy 
Forum (June 2018), https://fpf.org/wp-content/ 
uploads/2018/06/Beyond-Explainability.pdf. 








Co-Designing Checklists to Understand 
Organizational Challenges and Opportunities 
around Fairness in Al 

Michael A. Madaio, Luke Stark, Jennifer 
Wortman Vaughan, and Hanna Wallach, 

CHI ‘20: Proceedings of the 2020 CHI 
Conference on Human Factors in Computing 
Systems (April 2020): 1-14, https://doi. 
org/10.1145/3313831.3376445. 





Closing the Al Accountability Gap: Defining an 
End-to-End Framework for Internal Algorithmic 
Auditing 

Raji, |. D., Smart, A., White, R. N., Mitchell, M., 
Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, 
D., & Barnes, P., FAT* '20: Proceedings of the 
2020 Conference on Fairness, Accountability, and 
Transparency, (January 2020): 33-44, https://doi. 
org/10.1145/3351095.3372873. 





Supervisory Guidance on Model Risk 
Management 

US Federal Reserve Board (April 201 1), https:// 
www.federalreserve.gov/supervisionreg/srletters/ 


sr1107a1.paf. 





Understanding Artificial Intelligence Ethics 
and Safety: A Guide for the Responsible 
Design and Implementation of Al Systems 
in the Public Sector 

David Leslie, The Alan Turing Institute (2019), 
https://doi.org/10.5281/zenodo.3240529. 
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